Focused Crawling using Asynchronous Cellular Learning Automata

Web crawling is used to collect the web pages which will be indexed by a search engine. The search engine uses these crawled and indexed pages to answer users’ queries. Since the volume of web pages is very high and it increases continuously, search engines can index a limited number of web pages. Therefore, in recent years, the focused crawler algorithms have been introduced which act selectively during crawling and collect the web pages related to a specific topic. In this paper, an asynchronous cellular learning automata based approach for focused crawling is proposed. The proposed approach is a combination of web structure and web usage mining techniques and is composed of two phases. In the first phase the relationship structure of pages is determined using asynchronous cellular learning automata, hyperlinks and users’ behavior in visiting web pages, i.e. the related pages and their relevance degree are determined. In the second phase, the focused crawling is performed using the obtained relationship structure and the pages related to a specific topic are collected. Experimental results have shown the superiority of the proposed method (harvest rate and target recall) in comparison to Best First Crawler and its independency from initial set selection.